Skip to content

Arrow 57 / DataFusion 51 / Lance 2 + BlasGraph Algebra + SPO Triple Store#7

Merged
AdaWorldAPI merged 3 commits intomainfrom
claude/setup-adaworld-repos-4kPEX
Mar 13, 2026
Merged

Arrow 57 / DataFusion 51 / Lance 2 + BlasGraph Algebra + SPO Triple Store#7
AdaWorldAPI merged 3 commits intomainfrom
claude/setup-adaworld-repos-4kPEX

Conversation

@AdaWorldAPI
Copy link
Copy Markdown
Owner

PR: Arrow 57 / DataFusion 51 / Lance 2 + BlasGraph Algebra + SPO Triple Store

Title

feat: arrow 57, datafusion 51, lance 2 + BlasGraph semiring algebra + SPO triple store

Body


Follows up on #146 (closed — split was planned but the pieces are interdependent, so shipping as one clean PR rebased on current main).

Summary

Three additions in one PR because they share the dependency bump and build on each other:

  1. Dependency alignment — arrow 57, datafusion 51, lance 2, deltalake 0.30, pyo3 0.26
  2. BlasGraph — GraphBLAS-inspired sparse linear algebra over hyperdimensional bit vectors (3,173 lines, 81 tests)
  3. SPO triple store — Subject-Predicate-Object graph primitives with bitmap ANN, NARS truth gating, and Merkle integrity (1,443 lines, 38 tests)

All 119 new tests pass. Clippy clean across all crates. No breaking changes to existing APIs.


1. Dependency Upgrades

Dependency From To Crates
arrow / arrow-array / arrow-schema 56.2 57 lance-graph, catalog, python
datafusion (+ subcrates) 50.3 51 lance-graph, catalog, python
lance / lance-linalg / lance-namespace 1.x 2 lance-graph
deltalake 0.29 0.30 lance-graph
pyo3 0.25 0.26 lance-graph-python

What This Enables

With these three additions, lance-graph gains:

  • Graph algorithms as linear algebra — BFS, SSSP, PageRank expressed as semiring-parameterized matrix multiplications, extensible to custom algorithms by defining new semirings
  • Content-addressable triple storage — knowledge graph operations via bitmap fingerprints with sub-millisecond approximate matching
  • Confidence-gated queries — NARS truth values allow filtering unreliable edges before they enter the computation, reducing noise in multi-hop traversals
  • Integrity verification — Merkle stamping detects data corruption without full table scans

These compose naturally: the SPO store uses the HammingMin semiring from BlasGraph for chain traversal, and the bitmap fingerprints share the same bit-vector primitives as BlasGraph's BitVec type.


Migration Notes

  • No breaking changes to existing public APIs
  • Existing tests pass without modification
  • New graph module is additive — accessed via lance_graph::graph::{blasgraph, spo}
  • New dev dependency: blake3 (for Merkle hashing)
  • CI: added clippy + cargo check for lance-graph-python crate

Stats

New code:    ~5,000 lines of Rust
New tests:   119 (81 BlasGraph + 31 SPO unit + 7 SPO integration)
All passing: ✓
Clippy:      clean across all crates

claude added 3 commits March 13, 2026 10:25
…0.26

The arrow 57 upgrade caused a `links = "python"` conflict because
arrow-pyarrow 57 requires pyo3 0.26 while lance-graph-python used
pyo3 0.25 — two different pyo3-ffi versions in the same workspace.

Fix: bump pyo3 to 0.26 so all crates share one pyo3-ffi. Update
deprecated pyo3 0.26 APIs (with_gil→attach, allow_threads→detach,
PyObject→Py<PyAny>). Pyarrow zero-copy stays intact.

- Cargo.toml: restore lance-graph-python to members (remove exclude)
- lance-graph-python/Cargo.toml: pyo3 0.25 → 0.26
- executor.rs: update pyo3 0.26 API renames
- graph.rs: PyObject → Py<PyAny>

All crates: clippy clean, 423 tests pass.

https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
- style.yml: add clippy step for lance-graph-python
- build.yml: add cargo check for lance-graph-python

https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
Covers all 3 major additions: version upgrades, BlasGraph semiring
algebra, and SPO triple store — with architecture details, API
signatures, test matrix, and module dependency graph.

https://claude.ai/code/session_01Mcj8GxEtzmVba6RmuT7AjD
@AdaWorldAPI AdaWorldAPI merged commit c41fd71 into main Mar 13, 2026
AdaWorldAPI added a commit that referenced this pull request May 6, 2026
… MCP scope)

Same-day follow-up to MedCareV2#7 in the LanceProbe arc (#4#5 → R2-R6 → #7#8). Cannot fetch diff — MedCareV2 is outside the MCP allowlist; placeholder
entry preserves the trail. Two same-day MedCareV2 PRs are kept as separate
entries (do not collapse) per the appended note in the file.

If/when the diff is paste-shared or allowlist extended, promote Confidence from
"Cannot verify" to FINDING with concrete delta.
AdaWorldAPI pushed a commit that referenced this pull request May 6, 2026
…r VSA-scope correction

pattern.md (NEW, 578 lines) — usability patterns for traversing the
workspace's SoA/DTO surface as a graph (nodes = type defs, edges =
producer-consumer + duplicate + seam, subgraphs = clusters). 15 named
patterns covering canonical lookup, maturity scoring, Click-P-1 lens,
register-laziness check, dual-tier writes, ingestion-commit, lineage-
as-column, append-only governance, consult-first ordering, cross-
session blackboard via ledger row IDs, source-vs-claim divergence,
cluster-fix discipline, debug-as-API debt, scope-lock, and seam-naming.
Plus 7 critical findings and append-only update protocol.

ARCHITECTURE_ENTROPY_LEDGER.md (APPEND-only correction block) —

VSA scope correction per CLAUDE.md I-VSA-IDENTITIES iron rule:
- VSA-1 description tightened: Vsa16kF32 is for Markov chain over
  identity fingerprints exclusively. Provenance / JWT / RBAC / IDs
  are register territory, not VSA carriers.
- PERMUTE-1 description tightened: vsa_permute is unitary as an
  operation but the braiding usage is NOT lossless; cross-talk
  shrinks unbinding margin with N. Bound: N <= sqrt(d)/4.

New rows:
- EWA-SANDWICH-1 (PR #289 was missing from initial snapshot):
  Stage 3 / Smart / entropy 2. Scope: SPD-bounded propagation of
  cognitive Vsa16kF32 across Markov rho^d cycles. NOT a lineage
  error model (corrected from initial framing).
- SUBJECT-DTO-1: aspirational typed Subject struct with
  AuthSource enum (typed JwtClaims, not VSA). Implied by
  MedCareV2 #7+#8 wire shape.
- MOCK-DRIVER-1: q2 PR #35 Phase-3 placeholder, Stage 2.

Cross-repo resolution events:
- THINK-1 partial: q2 PR #35 dropped thinking-engine +
  cognitive-shader-driver deps from cockpit-server, migrated to
  canonical contract::cognitive_shader::*. Wire compression
  256x on cycle_fingerprint, 128x on color_acc.
- TRUTH-1 partial: q2 PR #35 cockpit-server bridges to
  lance-graph-planner::nars::truth::TruthValue::deduction.
- POLICY-1 + MEMBRANE-GATE-1: priority bump — MedCareV2 #8 now
  blocking on impl MembraneGate for Arc<rbac::Policy>.

Section G — ingestion-vs-traversal axis added: Cypher-parser path
(Option 1, ships now via PARSER-1 resolution) and splat-deposit path
(Option 2, gated by SPLAT-1) both converge on E1 typed Action API.

Retractions: E4 (VSA-bundled provenance) and E8 (geometrically-
bounded provenance via Vsa16kF32+EWA) — register laziness; do NOT
get appended to EPIPHANIES.md. E1, E2, E3, E5, E6, E7, E9 stand.

https://claude.ai/code/session_012AUf5NFgeAAQa5aQAKwSgx
AdaWorldAPI pushed a commit that referenced this pull request May 6, 2026
…r VSA-scope correction

pattern.md (NEW, 578 lines) — usability patterns for traversing the
workspace's SoA/DTO surface as a graph (nodes = type defs, edges =
producer-consumer + duplicate + seam, subgraphs = clusters). 15 named
patterns covering canonical lookup, maturity scoring, Click-P-1 lens,
register-laziness check, dual-tier writes, ingestion-commit, lineage-
as-column, append-only governance, consult-first ordering, cross-
session blackboard via ledger row IDs, source-vs-claim divergence,
cluster-fix discipline, debug-as-API debt, scope-lock, and seam-naming.
Plus 7 critical findings and append-only update protocol.

ARCHITECTURE_ENTROPY_LEDGER.md (APPEND-only correction block) —

VSA scope correction per CLAUDE.md I-VSA-IDENTITIES iron rule:
- VSA-1 description tightened: Vsa16kF32 is for Markov chain over
  identity fingerprints exclusively. Provenance / JWT / RBAC / IDs
  are register territory, not VSA carriers.
- PERMUTE-1 description tightened: vsa_permute is unitary as an
  operation but the braiding usage is NOT lossless; cross-talk
  shrinks unbinding margin with N. Bound: N <= sqrt(d)/4.

New rows:
- EWA-SANDWICH-1 (PR #289 was missing from initial snapshot):
  Stage 3 / Smart / entropy 2. Scope: SPD-bounded propagation of
  cognitive Vsa16kF32 across Markov rho^d cycles. NOT a lineage
  error model (corrected from initial framing).
- SUBJECT-DTO-1: aspirational typed Subject struct with
  AuthSource enum (typed JwtClaims, not VSA). Implied by
  MedCareV2 #7+#8 wire shape.
- MOCK-DRIVER-1: q2 PR #35 Phase-3 placeholder, Stage 2.

Cross-repo resolution events:
- THINK-1 partial: q2 PR #35 dropped thinking-engine +
  cognitive-shader-driver deps from cockpit-server, migrated to
  canonical contract::cognitive_shader::*. Wire compression
  256x on cycle_fingerprint, 128x on color_acc.
- TRUTH-1 partial: q2 PR #35 cockpit-server bridges to
  lance-graph-planner::nars::truth::TruthValue::deduction.
- POLICY-1 + MEMBRANE-GATE-1: priority bump — MedCareV2 #8 now
  blocking on impl MembraneGate for Arc<rbac::Policy>.

Section G — ingestion-vs-traversal axis added: Cypher-parser path
(Option 1, ships now via PARSER-1 resolution) and splat-deposit path
(Option 2, gated by SPLAT-1) both converge on E1 typed Action API.

Retractions: E4 (VSA-bundled provenance) and E8 (geometrically-
bounded provenance via Vsa16kF32+EWA) — register laziness; do NOT
get appended to EPIPHANIES.md. E1, E2, E3, E5, E6, E7, E9 stand.

https://claude.ai/code/session_012AUf5NFgeAAQa5aQAKwSgx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants